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TITLE OF THE INVENTION 
INFORMATION PROCESSING METHOD AND APPARATUS 
5 BACKGROUND OF THE INVENTION 

Field of the Invention 

[0001] The present invention relates to information 
processing technologies for performing various types of 
10 processing on multimedia content. 

Description of the Related Art 

[0002] In recent years, high-quality content made of 
images, audio, and the like and supplied as digital data has 
been supplied, and multimedia content including the mixed 
15 content has been circulated. In particular, due to the 

widespread use of the Internet, which is a global-scale 
network, in the past few years, such content can be easily 
available for pleasure. 

[0003] Furthermore, due to development in infrastructures 
20 for circulation and transmission of information, even 

individuals can transmit information. Also, transmission of 
various types of multimedia content including moving images, 
audio, and the like is becoming possible. In order to 
create such content, for example, technologies for easily 
25 editing and processing moving images photographed with video 
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cameras are required. 

[0004] In order to easily edit and process moving image 
data, audio data, and the like without troubling users, 
computers are required to automatically recognize content. 
5 However, it is naturally very difficult for computers to 

automatically recognize the details of information, such as 
moving images having a large amount of information that 
change over time. 

[0005] For document data or the like, which is one type 

10 of digital data, it is not difficult for computers to 

determine a part including a designated character or word. 
However, it is technologically very difficult for computers 
to understand the details expressed using moving images, 
audio, and the like. 

15 [0006] Thus, in order to easily recognize such multimedia 

content made of moving images, still images, audio, and the 
like, a method for providing, as metadata, linguistic 
information including the details and features of the 
multimedia content described using a keyword, an explanatory 

20 note, or the like or non-linguistic information including 

the features of the multimedia content described using an 
image or audio in a format that is easily processed by 
computers is adopted. Also, compatibility of the format of 
metadata itself is being achieved by means of MPEG-7 or the 

25 like. 
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[0007] In other words, in order to easily perform 
retrieval, editing, and the like on enlarged multimedia 
content, instead of understanding the details of the large 
content, the features and characteristics of the multimedia 
5 content are held inside or outside the content as metadata, 

and such information is used to realize a function to easily 
edit multimedia content. 

[0008] A central processing apparatus determining the 
details of content in accordance with the property and 

10 conditions by using an object (data) and a property 

(metadata) in order to perform load distribution in a case 
where an object-oriented data management system is used is 
disclosed as an example. The processing apparatus is 
capable of instructing appropriate processing to an 

15 appropriate processing part. 

[0009] However, using metadata in editing multimedia 
content, as described above, causes the problems described 
below. 

[0010] First, since a wide variety of metadata exists, 
20 all the information processing apparatuses cannot recognize 

metadata when editing multimedia content. Thus, if editing 
is performed on multimedia content without recognizing 
metadata, the metadata is not copied to the edited 
multimedia content. Consequently, metadata cannot be used 
25 for edited multimedia content. 
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[0011] Second, even if information processing apparatuses 
can recognize metadata when editing multimedia content, in a 
case where the multimedia content is changed in accordance 
with the editing, the metadata is inappropriate as metadata 
5 indicating a property of the changed multimedia content. 

Thus, the metadata cannot be correctly used for the edited 
multimedia content . 

[0012] In other words, the known technologies described 
above cause a problem in that metadata cannot be correctly 
10 used due to editing of multimedia content. 

SUMMARY OF THE INVENTION 

[0013] The present invention is designed in consideration 
15 of the above-mentioned problems, and the object of the 

present invention is to be capable of correctly using 
metadata even for edited multimedia content, irrespective of 
types of metadata. 

[0014] An information processing method according to the 
20 present invention for editing multimedia content including 

metadata includes a receiving step and a determining step. 
In the receiving step, operation information about an 
editing operation is received. In the determining step, it 
is determined whether or not the metadata is usable for the 
25 edited multimedia content in accordance with the operation 
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information and a property of the metadata in which a change 
in the reliability of the metadata due to the editing 
operation is written. 

[0015] Further objects, features and advantages of the 
5 present invention will become apparent from the following 

description of the preferred embodiments with reference to 
the attached drawings . 

BRIEF DESCRIPTION OF THE DRAWINGS 

10 

[0016] Fig. 1 is a block diagram showing the entire 
structure of an information processing system including an 
information processing apparatus according to a first 
embodiment of the present invention. 

15 [0017] Fig. 2 shows an example of a state in which the 

information processing apparatus according to the first 
embodiment reads a computer program, data, and the like from 
a medium storing the computer program and the like. 
[0018] Fig. 3 shows data written in XML, which is an 

20 example of metadata and meta-metadata used in the 

information processing apparatus according to the first 
embodiment . 

[0019] Fig. 4 shows a state in which editing and 
processing is being performed on video content in the 
25 information processing apparatus according to the first 



- 6 - 



embodiment . 

[0020] Fig. 5 is a flowchart showing the overview of the 
details and a procedure of a process performed for the 
metadata when a designated segment of the video content is 
5 edited and processed in the information processing apparatus 

according to the first embodiment. 

[0021] Fig. 6 shows calculation and determination results 
of each piece of the metadata when the metadata and the 
meta-metadata shown in Fig. 3 is processed in accordance 

10 with the procedure shown in Fig. 5 in the information 

processing apparatus according to the first embodiment. 
[0022] Fig. 7 shows data written in XML, which is an 
example of metadata and meta-metadata used in an information 
processing apparatus according to a third embodiment of the 

15 present invention. 

[0023] Fig. 8 shows data written in XML, which is an 
example of metadata and meta-metadata used in an information 
processing apparatus according to a fourth embodiment of the 
present invention . 

20 [0024] Fig. 9 is a flowchart showing the overview of the 

details and a procedure of a process performed for the 
metadata when a designated segment of video content is 
edited and processed in the information processing apparatus 
according to the fourth embodiment. 

25 [0025] Fig. 10 is a flowchart showing a process for 
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retrieving metadata in video content in the information 
processing apparatus according to the fourth embodiment. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

5 

[0026] An overview of the present invention will first be 
described. 

[0027] According to the present invention, metadata is 
provided to content, such as a moving image, and at the same 

10 time, a property of the metadata, in other words, the 

property indicating the extent of reduction in the 
reliability of the metadata when an operation such as 
editing is performed on the content is provided as meta- 
metadata. Both the metadata and meta-metadata can be used 

15 in each apparatus together. 

[0028] As a result of this, each piece of content 
includes meta-metadata, which is a redundant part. However, 
if the content is edited and processed, use of information 
of the meta-metadata enables each apparatus to determine 

20 whether or not the metadata is usable for the edited and 

processed content . 

[0029] If the metadata is determined to be unusable, the 
unusable metadata is prevented from being copied to the 
edited and processed content. Accordingly, unusable 
25 metadata is prevented from being added to content. Since 



- 8 - 



this determination is performed on the basis of meta- 
metadata, even an apparatus that cannot recognize the 
details of metadata can perform such determination. 
Furthermore, since such determination processing is 
5 performed in each apparatus, the processing operation can be 

decentralized. In other words, processing for metadata is 
determined in accordance with meta-metadata for the metadata 
in decentralized processing apparatuses. 
[0030] Also, the use of information of meta-metadata 
10 enables calculation of the reliability of metadata in edited 

content . 

[0031] As a result of this, for example, in a case where 
metadata is used for retrieval in content, in accordance 
with calculated reliability, metadata with low reliability 
15 is determined to be unusable and is excluded from the 

retrieved results. Accordingly, more accurate use of 
metadata can be achieved. 

[0032] Embodiments of the present invention will now be 
described with reference to the drawings. 

20 

First Embodiment 

[0033] Fig. 1 shows an example of the entire structure of 
an information processing system including an information 
processing apparatus according to a first embodiment of the 
25 present invention. 
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[0034] Referring to Fig. 1, a microprocessor (CPU) 11 
performs calculation, logical determination, and the like 
for various types of processing. The CPU 11 also controls, 
via an address bus AB, a control bus CB, and a data bus DB, 
5 components connected to each of the buses. The operations 

of the CPU 11 are instructed by programs stored on a ROM 12 
and a RAM 13. Also, a function of - the CPU 11 itself and 
mechanisms of computer programs enable a plurality of 
computer programs to be operated in parallel. 

10 [0035] The address bus AB transfers an address signal 

instructing a component to be controlled by the CPU 11. The 
control bus CB transfers and applies a control signal for 
each component to be controlled by the CPU 11. The data bus 
DB performs data transfer between components. 

15 [0036] The ROM 12 is a read-only permanent memory and 

stores a control program such as a processing program 
executed in the first embodiment. The ROM 12 also stores a 
computer program area and a data area in which procedures 
for control by the CPU 11 are stored. 

20 [0037] The RAM 13 is a writable random access memory. 

The RAM 13 is used as a computer program area and a data 
area in which procedures for control by the CPU 11 are 
stored and is also used as a temporary storage area for 
various computer programs and various types of data from 

25 each component other than the CPU 11. 
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[0038] Storage media, such as the ROM 12 and the RAM 13, 
store a computer program, data, and the like for realizing 
the first embodiment. The function of the first embodiment 
can be realized by reading and executing, by the system or a 
5 computer of the apparatus, program codes stored in these 

storage media. 

[0039] Also, there is a possibility that a recording 
medium storing the program and the data according to the 
present invention is supplied to the system or the apparatus 

10 and that the program is copied from the recording medium to 
a rewritable storage medium, such as the RAM 13. The 
recording medium may be a CD-ROM, a floppy disk, a hard disk, 
a memory card, a magnetic optical disk, or the like. The 
present invention can be realized by using any such 

15 recording medium. 

[0040] Referring to an example shown in Fig. 2, a 
computer program for realizing the first embodiment can be 
read from a recording medium 2-c which stores the computer 
program, where the recording medium 2-c storing the computer 

20 program is also included in the first embodiment. In Fig. 2, 

an apparatus 2-a, which is also shown in Fig. 1 as the 
information processing apparatus, includes a driver 2-b for 
reading the recording medium. 

[0041] Referring back to Fig. 1, a hard disk (DISK) 14 
25 functions as an external memory for storing various computer 
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programs, various types of data, and the like. The DISK 14 
contains a storage medium capable of reading and writing a 
large amount of information at relatively high speed, so 
that the DISK 14 can store and retrieve the various computer 
5 programs, the various types of data, and the like as 

necessary. Also, the stored computer programs and data can 
be called up completely or partially on the RAM 13 as 
necessary in accordance with an instruction from a keyboard 
or from various types of computer programs. 

10 [0042] A memory card (MemCard) 15 is a detachable storage 

medium. Storing information into the MemCard 15 and 
connecting the MemCard 15 to another apparatus causes the 
stored information to be referred to and transferred. 
[0043] A keyboard (KB) 16 includes various function keys, 

15 such as alphabet keys, hiragana keys, katakana keys, 

character symbol input keys for inputting a period and the 
like, cursor movement keys for instructing cursor movement, 
and the like. The KB 16 may be provided with a pointing 
device, such as a mouse. 

20 [0044] The details of a cursor register (CR) 17 is read 

and written by the CPU 11. A CRT controller (CRTC) 19 
displays a cursor in a position on a display device (CRT) 20 
corresponding to an address stored in the CR 17. 
[0045] A display buffer memory (DBUF) 18 stores a pattern 

25 of data to be displayed. 
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[0046] The CRTC 19 functions to display the details 
stored in the DBUF 18 on the CRT 20. 

[0047] The CRT 20 is formed by a cathode ray tube or the 
like. A dot display pattern and cursor display on the CRT 
5 20 is controlled by the CRTC 19. 

[0048] A character generator (CG) 21 stores patterns of 
characters and symbols to be displayed on the CRT 20. 
[0049] A communication device (NCU) 22 communicates with 
another computer apparatus or the like. Use of the NCU 22 

10 enables programs and data according to the first embodiment 

to be shared with the other computer apparatus. Referring 
to Fig. 1, the NCU 22 is connected to a personal computer 
(PC), a receiving, storing, and displaying apparatus (TV/VR) 
for receiving, storing, and displaying a television 

15 broadcast and an image photographed by a user, a game 

computer (GC) for home use, and the like via a network (LAN) 
for free information exchange. Needless to say, any 
apparatus can be connected to the apparatus according to the 
present invention via the network. Any type of network can 

20 be used. The network is not necessarily closed as shown in 

Fig. 1. The network may be connected externally. 
[0050] A receiving device (DTU) 23 realizes a receiving 
function of broadcast communication using a satellite or the 
like. The DTU 23 has a function to receive, by a parabolic 

25 antenna (ANT) , radio waves and the like broadcast via the 
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satellite in order to retrieve data being broadcast. Any 
types of satellite communication, such as broadcasting using 
ground waves, broadcasting using a coaxial cable or an 
optical cable, and distributing using the LAN or a large 
5 scale network, can be adopted. 

[0051] The information processing apparatus according to 
the first embodiment with the arrangement described above 
realizes a function to edit and process multimedia content, 
such as a moving image, a still image, and the like, and to 

10 perform an operation on metadata for the multimedia content. 

To begin with, the content and the metadata for the content 
are normally stored in a storage device, such as the DISK 14 
or the MemCard 15 shown in Fig. 1, or in an external storage 
device via the NCU 22 and the DTU 23 shown in Fig. 1. The 

15 content may be a moving image or a still image and may be 

recorded in any type of format. Similarly, metadata may be 
written in any format and recorded in any format as long as 
information (meta-metadata) according to the present 
invention is included. 

20 [0052] Fig. 3 shows metadata written in XML for video 

content, which is an example of multimedia content. The 
details of the metadata will be described. 

[0053] Metadata is described from a position 3-a. Since 
data written in such an XML format may include information 
25 about other than metadata, the description in the position 
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3-a is provided for distinction from the other information. 
The end of the description of the metadata is written in a 
position 3-y. This shows that the metadata is provided in 
an area from a position 3-b to a position 3-x. 
5 [0054] As the details of the metadata, to begin with, a 

metadata-provided segment is designated in the position 3-b. 
The metadata-provided segment from 0 hours, 2 minutes, 12 
seconds to 0 hours, 4 minutes, 34 seconds in the video 
content is written in the position 3-b, and the details of 
10 the metadata provided to the metadata-provided segment are 

written from the position 3-b. 

[0055] The end of the metadata-provided segment is 
written in the position 3-x. This shows that the metadata 
is provided in a segment up to a position 3-w. 

15 [0056] A description about a dominant color of a moving 

image part of the video content in the metadata-provided 
segment is provided as metadata in an area from a position 
3-c to a position 3-i. The start of the description of the 
metadata indicating the dominant color is written in the 

20 position 3-c and the end of the description of the metadata 
is written in the position 3-i. 

[0057] The dominant color is written in RGB levels in a 
position 3-h. The dominant color is used for retrieval in 
the video content and is written as a type of metadata. 
25 Meta-metadata according to the present invention is provided 
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in an area from a position 3-d to a position 3-g. The start 
of the description of the meta-metadata is written in the 
position 3-d and the end of the description of the meta- 
metadata is written in the position 3-g. 
5 [0058] The details of the meta-metadata are provided in 

the position 3-e. The meta-metadata indicates that the 
reliability of the metadata indicating the dominant color is 
reduced to 30% due to an operation of color change as an 
editing operation in the metadata-provided segment. 

10 Similarly, a description indicating that the reliability of 

the metadata indicating the dominant color is reduced to 0%, 
that is, there is no reliability of the metadata, due to cut 
and paste in video editing in the metadata-provided segment 
is provided in a position 3-f. 

15 [0059] Metadata indicating a dialog provided in the 

metadata-provided segment is provided in an area from a 
position 3-j to a position 3-o. The structure of the 
description is the same as in the structure of the 
description about the dominant color. "Hello" , which is the 

20 details of the dialog, is written in a position 3-n. Meta- 

metadata according to the present invention provided in a 
position 3-1 indicates that the metadata indicating the 
dialog becomes unreliable due to a change in audio 
information of the video content in the metadata-provided 

2 5 segment. 
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[0060] Similarly, a description about a photographed 
object of a moving image part of the video content in the 
metadata-provided segment is provided as metadata in an area 
from a position 3-p to the position 3-w. A description 
5 indicating that a person is photographed in the moving image 

in the metadata-provided segment and that the person's name 
is "Suzuki Ichiro" is provided in an area from a position 3- 
t to a position 3-v as metadata of the photographed object. 
Meta-metadata indicating that no operation changes the 

10 reliability of the metadata is provided in a position 3-r. 

[0061] According to the present invention, metadata is 
provided to content, such as a moving image, and at the same 
time, a property of the metadata is provided as meta- 
metadata, so that each apparatus can use both the metadata 

15 and meta-metadata. Thus, each piece of content includes 

meta-metadata, which is a redundant part. However, if the 
content is edited and processed, use of information of the 
meta-metadata enables each apparatus to determine whether or 
not the metadata is usable for the edited and processed 

20 content. Also, even an apparatus that cannot recognize the 
details of the metadata can perform such determination. 
[0062] Furthermore, since such determination processing 
is performed in each apparatus, the processing operation can 
be decentralized. In other words, processing for metadata 

25 is determined in accordance with meta-metadata for the 
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metadata in decentralized processing apparatuses. 
Accordingly, although there is no central processing part, 
processing can be decentralized. 

[0063] Fig. 4 shows a state in which editing and 
5 processing of video data is actually performed for video 

content. Fig. 4 shows an example of display on the CRT 20 
using the CRTC 19 and the CG 21 shown in Fig. 1 in a case 
where a window system is used. 

[0064] A part 4-1 is called a title bar and is provided 
10 for performing control of the entire window, such as moving, 

changing the size, and the like. A part 4-2 is called a 
tool bar and is provided for easily calling up main 
functions in the window. 

[0065] A part 4-3 is provided for selecting a moving 
15 image scene and a still image to be edited and processed. 

In Fig. 4, an operation for scenes in a moving image file 
named "Mr. Yamada 1 s Wedding .mpg" recorded using a video 
camera is shown. 

[0066] A desired scene can be selected by designating a 
20 desired segment of the moving image by operating a slider in 

a tab using an input device, such as the KB 16 shown in Fig. 
1. A picture surrounded by bold lines in the part 4-3 is a 
first one shot (one frame) of the selected scene. A picture 
of the preceding and following scenes and the like can be 
25 viewed in a similar manner. Accordingly, a user can select 
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a desired segment of video content for editing and 
processing as one scene. 

[0067] A part 4-4 is provided for playing and viewing the 
scene selected in the part 4-3. A desired shot and frame in 
5 the scene can be freely moved and viewed by operating a tab 

called a slider or the like. In a case where information 
about a photographing time of the scene and the like is 
provided, the information is displayed in the part 4-4. The 
user can acquire the details of the selected scene by 

10 viewing the part 4-4. 

[0068] A part 4-5 is provided for instructing editing, 
processing, effect, and the like for the segment selected in 
the parts 4-3 and 4-4. Selecting any one in a field for 
editing, processing, and effect and pressing an add button 

15 causes a designated processing operation (editing, 

processing, and effect) to be performed and the processed 
video data is additionally registered in a part 4-62. Here, 
processing for metadata using meta-metadata according to the 
present invention is performed. The details of the 

20 processing will be described below. 

[0069] A part 4-6 is provided for arranging the edited 
and processed video data to be displayed as video data. The 
video data to be output is stored in a state in which the 
video data is provided with metadata and meta-metadata as 

25 video content usable in the apparatus. A part 4-61 is 
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provided for designating an output format, and the part 4-62 
is provided for arranging video data created in the parts 4- 
3, 4-4, and 4-5. Here, the order of the final video content 
can be designated and changed by changing the order of the 
5 video data. 

[0070] A part 4-7 is provided for instructing to output 
the video data arranged in the part 4-6 as a piece of video 
content . 

[0071] Fig. 5 is a flowchart showing the overview of a 
10 procedure for a process when a segment of video content is 

designated as video data and an instruction for editing, 
processing and the like is provided. Fig. 5 shows an 
example of the process realized as computer programs. The 
programs may be operated partially or completely by an 
15 apparatus or a plurality of apparatuses. Alternatively, the 

programs may be operated in parallel at the same time. 
[0072] The details of the process and the procedure shown 
in Fig. 5 will now be described. In the first embodiment, 
when a segment is designated as video data and an 
20 instruction for editing, processing, and the like is 

provided (when operation information about an editing and 
processing operation is received) , the process is started in 
step S5-1. 

[0073] Step S5-2 prepares to use metadata to be processed 
25 in the process, meta-metadata for the metadata, and 
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information about the details of designated editing and 
processing . 

[0074] Step S5-3 retrieves metadata provided to a segment 
of the content to be edited and processed. 
5 [0075] Step S5-4 determines how much metadata is 

retrieved in step S5-3. If no metadata is retrieved, the 
process proceeds to step S5-6. 

[0076] If metadata is retrieved, the process proceeds to 
Step S5-5. In step S5-5, the reliability of each piece of 

10 metadata is calculated in accordance with its corresponding 
meta-metadata and the details of editing and processing, and 
it is determined whether or not the metadata should be 
provided to the cut-out video data. If the metadata becomes 
unusable due to editing and processing, the metadata is 

15 determined not to be copied. 

[0077] Fig. 6 shows the results of whether or not to keep 
(copy) each piece of metadata, the results being obtained by 
calculating the reliability of each piece of the metadata, 
when sepia processing for changing into an antique color 

20 tone like an old photograph and post-recording processing 

for rewriting audio information are performed as editing and 
processing operations in a segment including the metadata 
and the meta-metadata shown in Fig. 3. 
[0078] For sepia processing, although, generally, 

25 original metadata cannot be used as it is due to the entire 
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color tone change, a pattern of contrast remains even after 
the processing. Thus, the metadata is not completely 
unreliable after the processing. Consequently, although the 
reliability of metadata of a dominant color is reduced, the 
5 metadata is determined to be "keep" (copy), as shown in Fig. 
6. Also, since information about audio, a photographed 
object, and composition is not changed, metadata of a dialog 
and metadata of a photographed object are also determined to 
be "keep" (copy) . 

10 [0079] In contrast, for post-recording processing, 

although a moving image part is not changed, a dialog is 
changed or eliminated. Thus, metadata of the dialog cannot 
be used. Consequently, the metadata of the dialog is 
determined to be "do not keep" (do not copy) . Since 

15 metadata of a dominant color and metadata of a photographed 
object are reliable and usable, each piece of the metadata 
is determined to be "keep" (copy) . 

[0080] As described above, depending on the details of 
editing and processing, metadata is determined to be usable 

20 or unusable. Also, depending of the type of editing and 

processing, a value pattern and characteristics of metadata 
remain to some extent and the metadata can thus be utilized 
to some extent. In the first embodiment, even in such a 
case, determination of whether or not to copy metadata can 

25 be performed in accordance with the reliability. 
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[0081] Any procedure for determining whether or not to be 
usable in accordance with the calculated reliability is 
possible. In the first embodiment, the calculated 
reliability of metadata is compared with a predetermined 
5 reliability. If the calculated reliability of the metadata 

is greater than or equal to the predetermined reliability, 
the metadata is determined to be usable. 

[0082] Referring back to Fig. 5, step S5-6 determines 
metadata not provided in the segment cut out for editing and 
10 processing not to be copied so that the metadata is not kept 
in the cut-out video data. 

[0083] Step S5-7 provides only metadata that is 
determined to be kept in the segment of the video data cut 
out for editing and processing to the cut-out video data in 

15 accordance with the determination performed in steps S5-5 

and S5-6. The process is ended in step S5-8. 
[0084] As is clear from the above description, according 
to the first embodiment, if an operation, such as editing 
and processing, is performed on multimedia content provided 

20 with metadata, metadata that becomes unusable after editing 

and processing can be avoided from being copied to the 
multimedia content. Consequently, metadata can be correctly 
used for edited multimedia content. 

[0085] Also, for determination of usability of metadata, 
25 since information about the reliability of metadata after 
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editing and processing is written in meta-metadata, even an 
apparatus that cannot recognize the details of metadata can 
determine whether or not to copy the metadata to edited 
multimedia content in accordance with the reliability of the 
5 metadata. 



Second Embodiment 

[0086] In the first embodiment, a segment of video 
content is designated in order to perform editing and 

10 processing. For some types of metadata, the usability of 

metadata, in other words, the reliability of metadata after 
processing may be changed only by such an operation. 
[0087] For example, if only a part of a metadata-provided 
segment is designated as a segment of video content to be 

15 edited and processed, metadata of a dominant color and 
metadata of a dialog becomes unusable. 
[0088] In such a case, however, metadata of a 
photographed object remains usable. Thus, in a case where a 
part of video content is designated as a segment to be 

20 edited and processed, descriptions indicating that the 

reliability of the metadata of the dominant color and the 
metadata of the dialog is reduced to 0% are written in 
corresponding meta-metadata . Also, a description indicating 
that the reliability of the metadata of the photographed 

25 object is not changed is written in meta-metadata for the 
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metadata of the photographed object. As a result of this, 
unusable metadata is prevented from being copied to 
processed multimedia content. 

[0089] Also, for some types of metadata, the usability 
5 may be changed by the way of cutting out a front or back 

part. In such a case, however, using meta-metadata in which 
a change in the reliability of metadata due to such an 
operation is recorded enables calculation and determination 
of the usability of the metadata, as in the first embodiment. 

10 

Third Embodiment 

[0090] Although a change in the reliability of metadata 
due to an operation, such as editing, is written directly as 
meta-metadata in the first and second embodiments, the 

15 present invention is not limited to this. A storing 

destination for information about a change in the 
reliability of metadata due to editing may be written. A 
third embodiment of the present invention will now be 
described with reference to Fig. 7. 

20 [0091] Fig. 7 shows metadata written in XML for video 

content. The details of the metadata will be described. 
[0092] Metadata is described from a position 7-a. Since 
data written in such an XML format may include information 
other than about metadata, the description in the position 

25 7-a is provided for distinction from the other information. 
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The metadata is provided in an area from a position 7-b to a 
position 7-x. 

[0093] As the details of the metadata, to begin with, a 
metadata-provided segment is designated in the position 7-b. 
5 The metadata-provided segment from 0 hours, 2 minutes, 12 

seconds to 0 hours, 4 minutes, 34 seconds in the video 
content is written in the position 7-b, and the details of 
the metadata provided to the metadata-provided segment are 
written from the position 7-b. The end of the metadata- 
10 provided segment is written in the position 7-w. This shows 

that the metadata is provided in a segment up to a position 
7-v. 

[0094] A description about a dominant color of a moving 
image part of the video content in the metadata-provided 

15 segment is provided as metadata in an area from a position 

7-c to a position 7-h. The start of the description of the 
metadata indicating the dominant color is written in the 
position 7-c and the end of the description of the metadata 
is written in the position 7-h. 

20 [0095] The dominant color is written in RGB levels in a 

position 7-g. The dominant color is used for retrieval in 
the video content and is written as a type of metadata. 
Meta-metadata according to the present invention is written 
in an area from a position 7-d to a position 7-f. The start 

25 of the description of the meta-metadata is written in the 
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position 7-d and the end of the description of the meta- 
metadata is written in the position 7-f. 

[0096] The details of the meta-metadata is written in the 
position 7-e. A URL that stores information necessary for 
5 acquiring a change in the reliability of the metadata due to 
editing and processing in the metadata-provided segment of 
the video content and that functions as identification when 
the change of the reliability is acquired via a network is 
written in the position 7-e. In the present invention, 
10 information written here is not limited to a URL. Any type 

of information may be written as long as a storing 
destination for information about a change in the 
reliability of metadata is written. 

[0097] Metadata indicating a dialog provided in the 
15 metadata-provided segment is provided in an area from a 
position 7-i to a position 7-n. The structure of the 
description is the same as in the structure of the 
description about the dominant color. "Hello", which is the 
details of the dialog, is written in a position 7-m. Meta- 
20 metadata according to the present invention is provided in a 
position 7-k. Information necessary for acquiring 
information about a change in the reliability of the 
metadata due to editing and processing performed in the 
metadata-provided segment of the video content is written by 
25 URL in the position 7-k. 
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[0098] Similarly, a description about a photographed 
object of a moving image part of the video content in the 
metadata-provided segment is provided as metadata in an area 
from a position 7-o to the position 7-v. A description 
5 indicating that a person is photographed in the moving image 

in the metadata-provided segment and that the person's name 
is "Suzuki Ichiro" is provided in an area from a position 7- 
s to a position 7-u as metadata of the photographed object. 
Information necessary for acquiring information about a 

10 change in the reliability of the metadata due to editing and 

processing performed in the metadata-provided segment of the 
video content is written by URL in a position 7-q as meta- 
metadata according to the present invention. 
[0099] As is clear from the above description, writing a 

15 storing destination for information about a change in the 

reliability of metadata due to editing as meta-metadata 
allows advantages similar to those of the first embodiment. 

Fourth Embodiment 

20 [0100] A fourth embodiment of the present invention is 

characterized in that operation information about an 
operation, such as editing and processing, performed on 
multimedia content is added as an operation record. A 
specific example will now be described. 

25 [0101] Fig. 8 shows metadata written in XML for video 
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content, which is an example of multimedia content. The 
details of the metadata will be described. 

[0102] Metadata is described from a position 4-a. Since 
data written in such an XML format may include information 
5 about other than metadata, the description in the position 

4-a is provided for distinction from the other information. 
The end of the description of the metadata is written in a 
position 4-y. This shows that the metadata is provided in 
an area from a position 4-b to a position 4-x. 

10 [0103] As the details of the metadata, to begin with, a 

metadata-provided segment is designated in the position 4-b. 
The metadata-provided segment from 0 hours, 2 minutes, 12 
seconds to 0 hours, 4 minutes, 34 seconds in the video 
content is written in the position 4-b, and the details of 

15 the metadata provided to the metadata-provided segment are 

written from the position 4-b. 

[0104] The end of the metadata-provided segment is 
written in the position 4-x. This shows that the metadata 
is provided in a segment up to a position 4-w. 

20 [0105] A description about a dominant color of a moving 

image part of the video content in the metadata-provided 
segment is provided as metadata in an area from a position 
4-c to a position 4- j . The start of the description of the 
metadata indicating the dominant color is written in the 

25 position 4-c and the end of the description of the metadata 
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is written in the position 4-j . 

[0106] The dominant color is written in RGB levels in a 
position 4-i. The dominant color is used for retrieval in 
the video content and is written as a type of metadata. 
5 Meta-metadata according to the present invention is written 
in an area from a position 4-d to a position 4-h. The start 
of the description of the meta-metadata is written in the 
position 4-d and the end of the description of the meta- 
metadata is written in the position 4-h. 

10 [0107] The details of the meta-metadata are provided in 

the position 4-e. The meta-metadata indicates that the 
reliability of the metadata indicating the dominant color is 
reduced to 30% due to an operation of color change as an 
editing operation in the metadata-provided segment. 

15 Similarly, a description indicating that the reliability of 

the metadata indicating the dominant color is reduced to 0%, 
that is, there is no reliability of the metadata, due to cut 
and paste in video editing in the metadata-provided segment 
is provided in a position 4-f. 

20 [0108] A description not provided before an operation, 

such as editing and processing, performed on the multimedia 
content is provided in a position 4-g as a part of the meta- 
metadata. In other words, operation information about the 
editing and processing is added as an operation record when 

25 the editing and processing operation is performed on the 
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multimedia content. In the fourth embodiment, a description 
indicating that an operation of "ColorChange" (operation of 
changing a color) has been performed as an editing and 
processing operation is provided. 
5 [0109] Metadata indicating a dialog provided in the 

metadata-provided segment is provided in an area from a 
position 4-k to a position 4-q. The structure of the 
description is the same as in the structure of the 
description about the dominant color. "Hello", which is the 
10 details of the dialog, is written in a position 4-p. Meta- 

metadata provided in a position 4-m indicates that the 
metadata becomes unreliable due to a change in audio 
information of the video content in the metadata-provided 
segment . 

15 [0110] Also, an operation record indicating that an 

operation of "AudioChange" (operation of changing audio) has 
been performed is provided in a position 4-n, as in the 
operation record of the dominant color. 

[0111] Similarly, a description about a photographed 
20 object of a moving image part of the video content in the 

metadata-provided segment is provided as metadata in an area 
from a position 4-r to the position 4-w. A description 
indicating that a person is photographed in the moving image 
in the metadata-provided segment and that the person's name 
25 is "Suzuki Ichiro" is provided in an area from a position 4- 
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s to a position 4-v as metadata of the photographed object. 
Meta-metadata indicating that no operation changes the 
reliability of the metadata is provided in a position 4-t. 
[0112] Fig. 9 is a flowchart showing the overview of a 
5 procedure for a process when a segment of video content is 

designated as video data and an instruction for editing, 
processing and the like is provided. Fig. 9 shows an 
example of the process realized as computer programs. The 
programs may be operated partially or completely by an 

10 apparatus or a plurality of apparatuses. Alternatively, the 

programs may be operated in parallel at the same time. 
[0113] The details of the process and the procedure shown 
in Fig. 9 will now be described. In the fourth embodiment, 
when a segment is designated as video data and an 

15 instruction for editing, processing and the like is provided 

(when operation information about an editing and processing 
operation is received), the process is started in step S9-1. 
[0114] Step S9-2 prepares to use metadata to be processed 
in the process, meta-metadata for the metadata, and 

20 information about the details of designated editing and 

processing . 

[0115] Step S9-3 retrieves metadata provided in a segment 
of the content to be edited and processed. 
[0116] Step S9-4 determines how much metadata is 
25 retrieved in step S9-3. If no metadata is retrieved, the 
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process proceeds to step S9-6. 

[0117] Step S9-5 adds the details of the editing and 
processing operation, as an operation record, to meta- 
metadata in the metadata retrieved by step S9-4. The 
5 operation records provided in the positions 4-g and 4-n in 

Fig. 8 are added in step S9-5. 

[0118] Step S9-6 determines metadata not provided in the 
segment cut out for editing and processing not to be copied 
so that the metadata is not kept in the cut-out video data. 

10 [0119] Step S9-7 provides only metadata that is 

determined to be kept in the segment of the video data cut 
out for editing and processing to the cut-out video data in 
accordance with the determination performed in steps S9-5 
and S9-6. The process is ended in step S9-8. 

15 [0120] A process for retrieving metadata will now be 

described with reference to Fig. 10. In the fourth 
embodiment, when a segment is designated as video data and 
an instruction for an operation, such as retrieval, is 
provided, the process is started in step S6-1. Although an 

20 example of retrieval is described here, any type of 

operation for metadata is possible. 

[0121] Step S6-2 prepares for retrieval of metadata to be 
processed and meta-metadata for the metadata. 
[0122] Step S6-3 retrieves appropriate metadata in 
25 accordance with a segment and the type of metadata 
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designated by retrieval conditions. 

[0123] Step S6-4 determines how much candidate metadata 
for retrieval results is found in step S6-3. If no 
candidate metadata is found, the process proceeds to step 
5 S6-6. 

[0124] If candidate metadata for the retrieval results is 
found, the process proceeds to Step S6-5 to determine the 
usability of each piece of the metadata by calculating the 
reliability of the metadata in accordance with the 
10 corresponding meta-metadata . Metadata that is determined to 

be unusable is excluded from the candidate for the retrieval 
results . 

[0125] The details of step S6-5 will now be explained 
with reference to Fig. 8. In a case where multimedia 

15 content includes the metadata and the meta-metadata shown in 

Fig. 8, a description indicating that the reliability of 
metadata of a dominant color is reduced to 30% due to an 
operation of color change is provided as meta-metadata in 
the position 4-e and a description indicating that the 

20 operation of the color change has been performed is provided 

as an operation record in the position 4-g. 
[0126] In step S6-5, first, the reliability of the 
metadata at that time, in other words, the reliability of 
the metadata after performing color change is calculated in 

25 accordance with the operation record provided in the 
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position 4-g and with the information about the change in 
the reliability of the metadata provided in the position 4-e. 
[0127] Then, Step S6-5 also determines the usability of 
the metadata in accordance with the calculated reliability. 
5 This determination is performed by comparing the calculated 
reliability with predetermined threshold reliability. 
Metadata that is determined to be unusable is excluded from 
the retrieval results. 

[0128] Similarly, a description indicating that the 

10 reliability of metadata indicating a dialog is reduced to 0% 
due to an operation of audio change is provided as meta- 
metadata in the position 4-m in Fig. 8. Thus, although the 
reliability of the metadata indicating the dialog is not 
changed due to the operation of the color change, the 

15 reliability of the metadata indicating the dialog is reduced 

to 0% due to the operation of the audio change. Since the 
operation record indicating that the audio change has been 
performed is provided as the meta-metadata in the position 
4-n in Fig. 8, the reliability of the metadata indicating 

20 the dialog is calculated to be 0% and the metadata is 

determined to be unusable. As a result of this, the 
metadata is excluded from the retrieval results. 
[0129] Furthermore, since the metadata indicating the 
photographed object does not include meta-metadata about an 

25 operation record in Fig. 8, the metadata of the photographed 
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record is output as a retrieved result. 
[0130] As a more specific example, processing for 
retrieving metadata in video content on which sepia 
processing, that is, an operation of color change for 
5 changing into an antique color tone like an old photograph 

and post-recording processing for rewriting audio 
information are performed will be described. 
[0131] A case where the metadata of the dominant color, 
the metadata of the dialog, and the metadata of the 

10 photographed object shown in Fig. 8 are retrieved in 

accordance with predetermined conditions will be described. 
[0132] For sepia processing, although, generally, 
original metadata cannot be used as it is due to the entire 
color tone change, a pattern of contrast remains even after 

15 the processing. Thus, the metadata of the dominant color is 

not completely unreliable even after the processing and the 
reliability of the metadata after the processing is 30%. 
Accordingly, the metadata of the dominant color is 
determined to be usable. Also, since information about 

20 audio, a photographed object, and composition is not changed 

due to sepia processing, the metadata of the dialog and the 
metadata of the photographed object are also determined to 
be usable. Thus, for the video content subjected to the 
sepia processing, metadata of the dominant color, the 

25 metadata of the dialog, and the metadata of the photographed 
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object are output as retrieval results. 
[0133] In contrast, for post-recording processing, 
although a moving image part is not changed, a dialog is 
changed or eliminated. Thus, the reliability of the 
5 metadata of the dialog is reduced to 0% and the metadata is 

thus determined to be unusable. Since information about a 
color tone, a photographed object, and composition is not 
changed due to the post-recording processing, the metadata 
of the dominant color and the metadata of the photographed 

10 object are determined to be usable. Thus, for the video 

content subjected to the post-recording processing, the 
metadata of the dominant color and the metadata of the 
photographed object are output as the retrieved results. 
[0134] As described above, depending on the details of 

15 editing and processing, metadata in video content is 

determined to be usable or unusable. Also, depending of the 
type of editing and processing, a value pattern and 
characteristics of metadata remain to some extent and the 
metadata can thus be utilized to some extent. In the fourth 

20 embodiment, even in such a case, metadata can be excluded 

from retrieval results in accordance with the reliability. 
[0135] Referring now back to Fig. 10, the process is 
ended in step S6-6. 

[0136] As is clear from the above description, according 
25 to the fourth embodiment, operation information can be 
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provided as an operation record in meta-metadata when an 
operation, such as editing and processing, is performed on 
multimedia content including meta-metadata in which a change 
in the reliability of metadata due to editing and processing 
5 is written. 

[0137] Thus, when metadata is retrieved in edited and 
processed multimedia content, the reliability of the 
retrieved metadata is calculated, metadata with low 
reliability is determined to be unusable, and the unusable 

10 metadata is excluded from retrieval results. In other words, 

metadata can be correctly used in edited multimedia content. 
[0138] Also, by combining the fourth embodiment and the 
embodiments described above, when a segment of video content 
is designated and such an operation is written as an 

15 operation record in meta-metadata, metadata of a dominant 

color and metadata of a dialog are excluded from retrieval 
results if retrieval of the metadata is performed in the 
processed video content. 

2 0 Fifth Embodiment 

[0139] Although a change in the reliability of metadata 
due to an operation, such as editing, is written directly as 
meta-metadata in the fourth embodiment, the present 
invention is not limited to this. A storing destination for 

25 information about a change in the reliability of metadata 
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due to editing may be written. 

[0140] Writing a storing destination for information 
about a change in the reliability of metadata due to editing 
as meta-metadata allows advantages similar to those of the 
5 fourth embodiment. 

Other Embodiments 

[0141] The present invention is applicable to a system 
including a plurality of apparatuses, such as a host 

10 computer, an interface apparatus, a reader, and a printer. 

The present invention is also applicable to an apparatus, 
such as a copying machine or a facsimile machine. 
[0142] It is obvious that the object of the present 
invention can also be achieved by supplying a storage medium 

15 on which program codes of software for realizing the 

functions of the embodiments described above are recorded to 
a system or an apparatus and by causing a computer (or a CPU 
or an MPU) of the system or the apparatus to read and 
execute the program codes stored in the storage medium. 

20 [0143] In this case, the program codes themselves read 

from the storage medium realize functions of the embodiments 
described above. Thus, the storage medium that stores the 
program codes constitutes the present invention. 
[0144] The storage medium for supplying the program codes 

25 may be, for example, a floppy disk, a hard disk, an optical 
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disk, a magnetic optical disk, a CD-ROM, a CD-R, a magnetic 
tape, a nonvolatile memory card, a ROM, and the like. 
[0145] It is also obvious that the functions of the 
embodiments described above can be realized not only by 
5 executing the read program codes by the computer but also by 

performing part or all of the actual processing by an 
operating system (OS) or the like running on the computer on 
the basis of instructions of the program codes. 
[0146] It is also obvious that the functions of the 

10 embodiments described above can also be realized by 

performing part or all of the actual processing by a CPU or 
the like arranged in a function extension board of the 
computer or a function extension unit connected to the 
computer on the basis of instructions of program codes read 

15 from the storage medium after the program codes are written 

in a memory of the function extension board or the function 
extension unit. 

[0147] As described above, according to the present 
invention, even if multimedia content is edited, metadata 
20 can be correctly used for the edited multimedia content, 

irrespective of the type of metadata. 

[0148] Also, even if editing is performed on multimedia 
content including metadata, use of the correct metadata in 
the edited multimedia content is possible. 
25 [0149] While the present invention has been described 
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with reference to what are presently considered to be the 
preferred embodiments, it is to be understood that the 
invention is not limited to the disclosed embodiments. On 
the contrary, the invention is intended to cover various 
5 modifications and equivalent arrangements included within 
the spirit and scope of the appended claims. The scope of 
the following claims is to be accorded the broadest 
interpretation so as to encompass all such modifications and 
equivalent structures and functions. 



